1. Technical Field
The present technology relates to a storage medium having stored therein a stereoscopic image display program for stereoscopically displaying an image, a stereoscopic image display device, a stereoscopic image display system, and a stereoscopic image display method.
2. Description of the Background Art
There are conventional devices for displaying a plurality of images captured by a plurality of cameras on a stereoscopic display, thereby stereoscopically displaying an image. For example, Japanese Laid-Open Patent Publication No. 2003-264851 describes displaying two images, a right-eye image and a left-eye image, on each other so that the right eye of the user sees the right-eye image and the left-eye sees the left-eye image by using the parallax barrier method or the lenticular method. Thus, the user can stereoscopically view the image.
Note that in a method where stereoscopic display is produced by displaying two images on each other as described in Japanese Laid-Open Patent Publication No. 2003-264851, the zero-parallax distance in stereoscopic display (the distance from the point of view to the display object as seen on the display plane in stereoscopic display) can be changed by changing the positional relationship between the two images. FIG. 24 is a table showing the relationship between the positional relationship between the two images and the zero-parallax distance. Where the positional shift between a left-eye image 91 and a right-eye image 92 is relatively large as shown in the upper section of the table shown in FIG. 24, the zero-parallax distance is relatively far (a more distant object appears in focus). On the other hand, where the positional shift between the left-eye image 91 and the right-eye image 92 is relatively small as shown in the lower section of the table shown in FIG. 24, the zero-parallax distance is relatively near.
With the method described above, the image of the portion where the two left and right images lie on each other (the area surrounded by hatching in FIG. 24) appears stereoscopically, and the image where they do not lie on each other does not appear stereoscopically. That is, with the method described above, the size of the area that appears stereoscopically varies when the zero-parallax distance is changed.
Here, assuming that an area of a certain range of the two images is displayed on the screen, a portion where the two images do not lie on each other may be displayed depending on the zero-parallax distance. For example, in FIG. 24, assume that an area 93 surrounded by the dotted line is the area to be displayed on the screen. Then, where the zero-parallax distance is far as shown in the upper section of the table shown in FIG. 24, only the portion where two images lie on each other is included in the area 93, and therefore the entire image displayed on the screen appears stereoscopically. However, where the zero-parallax distance is near as shown in the lower section of the table shown in FIG. 24, portions where the two images do not lie on each other are included in the area 93, and therefore portions of the image displayed on the screen do not appear stereoscopically. In this case, an image appearing stereoscopically and an image not appearing stereoscopically are displayed on the screen, resulting in an image that is not easy to view. Note that it is possible to use a method in which no image is displayed (black is displayed) in portions where the two images do not lie on each other. With this method, however, the width of the displayed image is narrowed when the zero-parallax distance is near, thereby giving an awkward feel to the user and failing to make effective use of the display screen because of the area where nothing is displayed.
Thus, a feature of the present technology is to provide a storage medium having stored therein a stereoscopic image display program, a stereoscopic image display device, a stereoscopic image display system, and a stereoscopic image display method, with which images can be displayed in a manner easy to view in stereoscopic display.
In order to solve the above, the present technology employs the following configurations (1) to (8).
(1)
The present technology is a computer-readable storage medium having stored therein a stereoscopic image display program to be executed by a computer of an information processing device which displays an image on a stereoscopic display device capable of producing stereoscopic display using two captured images captured by two image-capturing devices. The stereoscopic image display program instructs the computer to function as a shift amount determination section, an area size changing section, and a display control section. The shift amount determination section determines a shift amount from a reference position for adjusting parallax in stereoscopic display for an area of a predetermined size representing a display range of each captured image. For at least one of the areas, if the area moved by the shift amount protrudes out of the captured image, the area size changing section changes the size of the area so that the area does not protrude out of the captured image. The display control section displays an image in a display object area of each of the two captured images in a predetermined area on a screen of the stereoscopic display device, wherein the display object area is the area determined by the shift amount determination section and the area size changing section.
The “stereoscopic display device” may be any device which produces stereoscopic display by making the left eye and the right eye of the user recognize images different from each other. The “stereoscopic display device” may be, for example, a display device of a parallax barrier type, a lenticular type, or the like, or may be a display device including a display for the left eye and a display for the right eye, such as a head-mounted display. The “stereoscopic display device” may be a display device with which the left eye and the right eye are made to recognize images different from each other with the user using a special pair of glasses.
The “information processing device” is a concept including any computer that performs an information process by executing a computer program, in addition to a game device as illustrated in the embodiment to be described later. The “information processing device” may or may not be of a portable type, and it may be provided integral with, or separate from, the stereoscopic display device.
Although an example of the “stereoscopic image display program” is a game program as illustrated in the embodiment to be described later, it is a concept including an application program to be executed in a personal computer or a portable terminal.
The “predetermined size” may be any size and may be a range larger than the captured image. The “reference position” may be any position on the captured image.
The “predetermined area on a screen” may be the entire area of the screen or a partial area of the screen.
With the configuration (1) above, the size of the area representing the display range of the captured image is changed so that the area does not protrude out of the captured image, and therefore the captured image is always included across the entire area of the display area. Here, in a case where the captured image is not included in a portion of the display area, stereoscopic display is not produced for that portion, and therefore the display may not be easy to view and the user may experience an awkward feel. In contrast, with the configuration (1) above, the captured image is always included across the entire area of the display area, and it is therefore possible to prevent a display that is not easy to view and it is possible to produce stereoscopic display in a manner easier to view.
(2)
The stereoscopic image display program may instruct the computer to further function as a recognition section for performing, for each of the captured images, a process of recognizing a predetermined image-capture object included in the captured image. Then, the shift amount determination section determines the shift amount so that the moved area represents a display range such that a zero-parallax distance in stereoscopic display is a distance in accordance with a distance from the image-capturing device to the image-capture object.
The “predetermined image-capture object” may be any object as long as it can be recognized by the recognition process, in addition to a marker 53 in the embodiment to be described later. For example, the face of the user (player), etc., may be recognized as the predetermined image-capture object.
Although the “zero-parallax distance in stereoscopic display being a distance in accordance with a distance from the image-capturing device to the image-capture object” means that the zero-parallax distance is determined by the distance from the image-capturing device to the image-capture object, it does not mean that the “distance from the image-capturing device to the image-capture object” is calculated. That is, the “display position determination section” may calculate the distance (virtual marker distance) corresponding to the distance from the image-capturing device to the image-capture object (step S13), calculate the zero-parallax distance in accordance with the calculated distance (step S17), and determine the display position in accordance with the zero-parallax distance (S21), as in the embodiment to be described later. The “display position determination section” may perform matching between the two captured images with respect to the predetermined image-capture object to determine the display position so that the position of the image-capture object is the same between the two captured images as described in (Variation where zero-parallax distance is not calculated) to be described later.
With the configuration (2) above, the display areas of the two images displayed on the stereoscopic display device are adjusted so that the zero-parallax distance in stereoscopic display is a distance in accordance with the distance from the image-capturing device to the predetermined image-capture object. That is, the zero-parallax point of the image to be stereoscopically displayed is automatically adjusted according to the distance from the image-capturing device to the predetermined image-capture object. Then, automatic adjustment can be made so that the zero-parallax point is set at the position of the predetermined image-capture object or at a position in the vicinity of the image-capture object, for example, and it is therefore possible to produce stereoscopic display in which the zero-parallax point is set at a virtual object displayed in the vicinity of the image-capture object. That is, it is possible to produce stereoscopic display in which the zero-parallax point is set at the virtual object even if the distance from the image-capturing device to the image-capture object varies. Therefore, with the configuration (2) above, in a case where an image of a virtual object produced by using an augmented reality technique is stereoscopically displayed, it is possible to display the image of the virtual object in a manner easy to view.
If the change to the size of the area by the area size changing section is not made in a case where the display area is changed automatically irrespective of the user's intention as with the configuration (2) above, the display area is changed, without a permission of the user, so that the captured image is no longer included in a portion of the display area. Therefore, the user may feel that the image suddenly becomes uneasy to view and particularly experience an awkward feel. Therefore, in a case where the display area is changed automatically irrespective of the user's intention as with the configuration (2) above, it is particularly effective to automatically adjust the size of the display area by the present invention.
(3)
The recognition section may calculate a positional relationship between the predetermined image-capture object and each image-capturing device based on the captured image. Then, the stereoscopic image display program instructs the computer to further function as a distance calculation section for calculating a length corresponding to a distance from the image-capturing device to the image-capture object based on at least one of the positional relationships calculated by the recognition section. The shift amount determination section determines the shift amount based on the zero-parallax distance determined by a distance calculated by the distance calculation section.
The “positional relationship” may be any information which reflects the positional relationship between the image-capturing device and the predetermined image-capture object, and it may be the position and the orientation of the image-capturing device with respect to the predetermined image-capture object or the position and the orientation of the predetermined image-capture object with respect to the image-capturing device. The “positional relationship” may be the position and the orientation of the virtual camera (view matrix) calculated in step S2 of the embodiment to be described later.
With the configuration (3) above, since the length corresponding to the distance from the image-capturing device to the image-capture object is actually calculated and the zero-parallax distance is determined based on this length, it is possible to accurately set the zero-parallax distance. For example, this makes it easy to set the zero-parallax distance to be the distance from the image-capturing device to the image-capture object, or to set the zero-parallax distance to be a distance that is nearer (or farther) by a predetermined distance with respect to the distance from the image-capturing device to the image-capture object.
(4)
The display control section may perform, for each captured image, a process of synthesizing an image of a virtual object produced by using the positional relationship with the captured image present in the display object area to display two images obtained by synthesis on the stereoscopic display device.
With the configuration (4) above, it is possible to stereoscopically display a synthesized image obtained by synthesizing together a captured image and a virtual image.
(5)
The display control section may include a camera position calculation section, a camera viewing field calculation section, and a virtual image production section. The camera position calculation section calculates a position and an orientation of each of two virtual cameras in a virtual space where the virtual object is placed based on the corresponding positional relationship. The camera viewing field calculation section calculates a viewing field range (the view volume, the projection matrix) of each of the virtual cameras so that the virtual space corresponding to a real space in the display object area is the range. The virtual image production section produces an image of the virtual object to be synthesized with each captured image based on the position, the orientation and the viewing field of the corresponding virtual camera.
With the configuration (5) above, the virtual cameras are set so that the virtual space corresponding to the display object area of the captured image is the viewing field range. Then, even when the size of the display area of the captured image is changed as in the present invention, the range of the captured image and the range of the virtual image can be made to correspond to each other, and it is possible to display an image of the virtual object at an appropriate position on the captured image.
(6)
The camera viewing field calculation section may calculate the viewing field range of the virtual camera so that a positional relationship of the display object area with respect to an entire area of the captured image corresponds to a positional relationship of the viewing field range to be calculated with respect to the viewing field range of the virtual camera corresponding to the viewing field range of the image-capturing device.
In the example of FIG. 22, the “positional relationship of the display object area with respect to the entire area of the captured image” is the positional relationship of a display area 63 with respect to a left captured image 55. In the example of FIG. 22, the “viewing field range of the virtual camera corresponding to the viewing field range of the image-capturing device” is the viewing field range represented by the straight lines L3 and L4. That is, in the example of FIG. 22, the “positional relationship of the viewing field range to be calculated with respect to the viewing field range of the virtual camera corresponding to the viewing field range of the image-capturing device” is the positional relationship of the viewing field range represented by the straight lines L5 and L6 with respect to the viewing field range represented by the straight lines L3 and L4.
With the configuration (6) above, since the viewing field range of the virtual camera is calculated by using the positional relationship between the area of the captured image and the display object area, it is possible to easily calculate the viewing field range corresponding to the display object area.
(7)
When an area moved by the shift amount protrudes out of the captured image, the area size changing section may shrink the area so that a protruding vertex of the area is at a position on a circumference of the captured image and so that a center position thereof is not changed.
With the configuration (7) above, it is possible to minimize the shrinking of the area which protrudes out of the captured image. Therefore, it is possible to prevent the range displayed on the screen from being excessively small (excessive zooming in), and it is possible to display an image easier to view.
(8)
The shift amount determination section may successively determine a shift amount for a predetermined captured image to be displayed on the stereoscopic display device among captured images successively obtained from the two image-capturing devices. Then, each time the shift amount is determined for the predetermined captured image, the area size changing section determines whether an area moved by the shift amount protrudes out of the captured image, and changes a size of the area if the area protrudes out of the captured image. The display control section successively displays an image in each display object area of the predetermined captured image in the predetermined area on the screen of the stereoscopic display device.
The “predetermined captured image” may be any image, and may for example be a captured image for which the recognition process has succeeded or all captured images successively obtained from the image-capturing device. That is, of the captured images successively obtained from the image-capturing device, all of the captured images may be displayed successively, or only the captured images for which the recognition process has succeeded may be displayed. For example, in a case where the next recognition process is performed for a captured image that is the latest at the point in time when the recognition process for a captured image has been completed, only captured images subjected to the recognition process may be displayed.
With the configuration (8) above, for a predetermined captured image, it is successively determined whether the area protrudes out of the captured image, and if it does, the size of the area is changed. As a result, captured images are displayed while successively changing the display area to an appropriate size. Therefore, for captured images captured in real time, it is possible to produce stereoscopic display while adjusting the size of the display area in real time.
The present technology may be carried out in the form of a stereoscopic image display device including equivalent sections to the sections described above. In this stereoscopic image display device, the sections may be implemented by a computer executing a stereoscopic image display program, or some or all of the sections may be implemented by dedicated circuits. The present invention may be carried out in the form of a stereoscopic image display system having one or more information processing device including the sections described above. Then, the one or more information processing device may directly communicate via wired or wireless communication or may communicate via a network. Moreover, the present invention may be carried out in the form of a stereoscopic image display method performed by the sections described above.
With the present technology, the size of the area representing the display range of the captured image is changed so that the area does not protrude out of the captured image, and therefore the captured image is always included across the entire area of the display area. Therefore, it is possible to prevent a display that is not easy to view and it is possible to produce stereoscopic display in a manner easier to view.
These and other objects, features, aspects and advantages of the present technology will become more apparent from the following detailed description of the present technology when taken in conjunction with the accompanying drawings.